Skip to content

Store one cliff result per simulation run in the v2 database, with grants - #4327

Draft
anth-volk wants to merge 12 commits into
masterfrom
v2plan/c1-37-cliff-results
Draft

anth-volk wants to merge 12 commits into
masterfrom
v2plan/c1-37-cliff-results

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4326

Summary

Adds revision bc334f9877b7 to the v2 chain (on 9d1d12a7ac58, autogenerated with --rev-id). It creates cliff_impacts: one row per cliff calculation of a simulation run, with the calculation's state and, once it succeeds, the run's cliff gap and cliff share. It also declares the table's grants.

Design decisions

The simulation API's change that writes the table and its change that starts calculations agreed this design before the revision was built:

  • One result per run, linked to the run, not a report. Every report naming the run shares the result. Cliffs need marginal tax rates, which recalculate each household with extra earnings, so a shared run never stores the cliff columns, and US national runs keep their region-group split. A request asking for cliffs starts a separate calculation of the run.
  • The row records the calculation's state: status (the v2_simulation_status type runs use), modal_call_id, error_message (code: message), started_at and completed_at. This lets a request tell "no result" from "being calculated" from "done".
  • At most one calculation per run that hasn't failed (uq_cliff_impacts_unfailed, partial unique on simulation_run_id where status <> 'failed', also declared for SQLite).
    • The simulation service inserts with ON CONFLICT (simulation_run_id) WHERE status <> 'failed' DO NOTHING, so two requests at once start one calculation.
    • A failed calculation never counts, as for runs: it stays as a record, and the next request inserts another.
    • Rejected: one row per run that a retry resets. That would need wider grants, and a stale call could write onto the reset row.
  • Checks: a succeeded or failed row has completed_at; a succeeded row has both numbers, and no other row has either. The numbers aren't range-checked: .py doesn't keep the share to at most 1, and a filtered simulation with no adults gives NaN.
  • CASCADE on the run, as poverty and inequality have. The result is part of its run, and nothing references it.
  • No version columns. A result inherits its run's catalog and package versions, so it is calculated only for a run on the versions the simulation API serves.
  • An index on (status, started_at), so the simulation API's stuck-run sweep can fail a calculation whose worker ended without an outcome.

Grants (policyengine_api/data/v2/privileges.py)

Role Privileges on cliff_impacts
policyengine_v2_simulation_service SELECT, INSERT, UPDATE (status, modal_call_id, error_message, completed_at, updated_at): insert a calculation, record its Modal call, or fail it when Modal refuses the call
policyengine_v2_modal_worker SELECT, UPDATE (status, modal_call_id, error_message, started_at, completed_at, updated_at, cliff_gap, cliff_share): claim, finish or fail; no INSERT
policyengine_v2_runtime SELECT
policyengine_v2_catalog_writer, policyengine_stage12_<environment> none

Neither simulation API role may change the run a row names or delete a row. The deploy's existing reconcile_v2_privileges.py apply and verify steps grant and check these after the schema upgrade; no role needs creating.

The revision

  • Post-generation correction: status reuses the baseline's v2_simulation_status type (postgresql.ENUM(..., create_type=False)), as 1356f658347d does for simulation_runs.status. The generated create_table would create the type again and fail. This is the only correction, and it is commented in the revision.
  • Downgrade: drops only the table, with its rows, and leaves the type to simulation_runs.
  • Compatibility: it writes no data and changes no other table. API v1's previous release doesn't map the table, so it keeps working at this revision.
  • Head: V2_ALEMBIC_HEAD_REVISION moves to bc334f9877b7.
  • Out of scope here:
    • report_runs' four cliff columns stay until a later revision drops them.
    • API v1 doesn't read the table yet.
    • Until the simulation API writes the table, no run has a cliff calculation.
  • Parent moved after Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies #4325: the revision was generated on f519a77ba122. Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies #4325's dynamics revision 9d1d12a7ac58, on the same parent, merged first, so this PR merges master and now revises 9d1d12a7ac58. Generating it again from there gives the same operations.

Docs

  • docs/engineering/skills/alembic-migrations.md: a section for this revision, plus the two simulation roles' rows in the role table.
  • The downgrade chains in that file, docs/migration/stage-9-v2-metadata.md and docs/migration/stage-12a-user-login.md.

CI for #4325's dynamics tests

#4325 added two PostgreSQL modules that no CI job ran. One of its tests broke when this revision moved onto the dynamics revision, and CI didn't catch it.

  • test_alembic_v2_dynamics.py::test_the_downgrade_fails_changing_nothing_while_a_dynamic_is_stored now downgrades to the dynamics revision instead of upgrading to it, because the head is now later than that revision. It also checks that the failed downgrade leaves legacy_policy_mappings.dynamic_id in place.
  • .github/scripts/test_alembic_v2_lifecycle.sh now runs test_alembic_v2_dynamics.py. These tests downgrade and upgrade the schema, so they run in the V2 Postgres migration lifecycle job, with its own database, like every other revision's migration tests.
  • v2-integration-check.yml's policy persistence step now also runs test_v2_dynamic_persistence.py, after the managed roles get their privileges, since it runs as API v1's runtime role.
  • The dynamics migration tests' engine fixture now checks that the target is the disposable database before upgrading it, as the cliff impacts fixture does.
  • tests/unit/test_alembic_workflows.py requires both modules where they now run, and that the migration tests stay out of the integration job.

Testing

Test-first: the first commit holds the failing tests, seen failing because there was no CliffImpact model, no cliff_impacts grants and no revision bc334f9877b7. The tests below were run by node ID against a disposable PostgreSQL 17 database.

  • Unit:
    • test_cliff_impact_models covers the columns, checks and indexes, and checks that SQLite keeps one unfailed calculation per run and refuses numbers on any row but a succeeded one.
    • The privilege manifest's tests for both simulation roles, the runtime role and the new table.
    • The revision chain, and this revision's operations and its one correction.
    • The v2 lifecycle script's test list.
    • test_simulation_run_models' per-simulation result tables.
  • Integration (test_alembic_v2_cliff_impacts, PostgreSQL):
    • the revision adds only its table and no enum type;
    • the downgrade leaves the type, and upgrading again works;
    • two inserts at once start one calculation, and a failed one is retried;
    • the checks, and the CASCADE.
  • Integration (test_v2_simulation_role_privileges, as the managed roles):
    • the simulation service inserts, joins, records a call and fails a calculation, and is refused claiming, writing numbers, changing the run or deleting;
    • the Modal worker claims and finishes, and is refused inserting, changing the run or deleting;
    • the runtime role reads a report run's results and is refused writing;
    • the catalog writer and the Stage 12 role can't read;
    • each simulation role can update exactly its declared columns.
  • Integration (test_alembic_v2_simulation_runs): the head foreign-key test now counts cliff_impacts among the simulation side's tables.
  • Checks:
    • alembic -c alembic-v2.ini check: no drift;
    • reconcile_v2_privileges.py apply --create-missing-roles and verify: pass;
    • also passing: lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py, ruff, ruff format and mypy.
  • Explicit mypy on the changed policyengine_api/data/v2/models files reports the Field(sa_type=...) overload error on the new status, started_at and completed_at fields. That is the same error the existing SimulationRun fields have on master; the directory is outside make typecheck-v2.
  • Dynamics tests: the five tests of each module pass by node ID against PostgreSQL 17, and so do the three workflow tests, which failed first.
  • Environment variables: the CI changes add test files to .github/scripts/test_alembic_v2_lifecycle.sh and to one step of v2-integration-check.yml.
    • The lifecycle script's tests read V2_MIGRATION_DATABASE_URL and V2_ALEMBIC_DISPOSABLE_TEST. alembic-v2-check.yml sets both as literal job-level env: values, for the job's own PostgreSQL service.
    • test_v2_dynamic_persistence.py reads the same two, plus DEPLOYMENT_ENVIRONMENT for the managed roles' names, which defaults to test. v2-integration-check.yml sets all three as literal job-level env: values, for its own PostgreSQL service.
    • The deploy steps that apply the revision and grants are unchanged: seed-v2-database.yml, with V2_MIGRATION_DATABASE_URL from each GitHub environment's secrets.
  • No inline suppressions added.

🤖 Generated with Claude Code

anth-volk and others added 3 commits November 17, 2026 17:36
Each simulation run can have one cliff calculation that hasn't failed,
holding its state and, once it succeeds, the run's cliff gap and cliff
share. The tests cover the model, its checks and partial unique index,
the revision that adds the table, and the grants: the simulation service
starts calculations, the Modal workers finish them, the Public API reads
them, and the catalog writer and the Stage 12 role can do neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cliff_impacts (revision bc334f9877b7, autogenerated on f519a77ba122):
one row per cliff calculation of a simulation run, with its state and,
once it succeeds, the run's cliff gap and cliff share. A partial unique
index allows one calculation per run that hasn't failed, so identical
requests join it and a failed one never counts. The row is deleted with
its run, and inherits the run's versions. Its status reuses
v2_simulation_status, the revision's one post-generation correction.

The simulation service may insert calculations and record or fail their
Modal calls, the Modal workers may write their state and results, the
Public API may read them, and no other role may touch them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Stage 9 and Stage 12A runbooks' downgrade chains now start with
bc334f9877b7, and the head foreign-key test counts cliff_impacts among
the simulation side's tables. The changelog fragment and the skill
section say what each role may do, since the simulation API writes the
table only once its own change lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.66%. Comparing base (003ee49) to head (22616f1).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4327      +/-   ##
==========================================
+ Coverage   93.63%   93.66%   +0.03%     
==========================================
  Files         408      408              
  Lines       23282    23297      +15     
  Branches     3648     3648              
==========================================
+ Hits        21799    21822      +23     
+ Misses        868      858      -10     
- Partials      615      617       +2     
Flag Coverage Δ
integration 58.06% <100.00%> (+0.26%) ⬆️
unit 90.93% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The v2 metadata now holds 51 tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The changelog fragment, the migrations skill, the revision's docstring and
two test docstrings now name the service API v1, and the test of its reads
is renamed to match. The revision's schema and grants are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
Brings in ddc422c, which changes #4327's wording from "the Public API"
to "API v1" and renames one privilege test. No file this branch changes
is touched, and revision bc334f9877b7 is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dynamics revision 9d1d12a7ac58 (#4325) reached master first on the same
parent, so the cliff impacts revision bc334f9877b7 now revises it, and stays
the v2 head. Generating it again from 9d1d12a7ac58 gives the same operations.
The revision tests list both in chain order, the cliff impacts integration
test downgrades to the dynamics revision, and the migrations skill and the
stage 9 and stage 12a downgrade notes name the dynamics revision's step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 20, 2026 01:13
The cliff impacts revision now follows the dynamics revision, so the test's
upgrade to the dynamics revision from the head did nothing, and its failing
downgrade then rolled back the cliff impacts step too, leaving the database at
bc334f9877b7. A downgrade to the dynamics revision restores what the test
checks: the dynamics downgrade fails and changes nothing while a dynamic is
stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The failed downgrade leaves legacy_policy_mappings.dynamic_id in place, which
the dynamics downgrade drops before the step that fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 3 commits November 20, 2026 01:29
The V2 Postgres migration lifecycle job is to run test_alembic_v2_dynamics.py,
and the cross-database integration job test_v2_dynamic_persistence.py, after
the managed roles get their privileges, since it runs as the runtime role.
Both came with #4325, and no CI job runs either yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The V2 Postgres migration lifecycle script now runs test_alembic_v2_dynamics.py
after the cliff impacts module, against its own disposable database, and the
cross-database integration job runs test_v2_dynamic_persistence.py in its
policy persistence step, after the managed roles get their privileges. Both
read only V2_MIGRATION_DATABASE_URL, V2_ALEMBIC_DISPOSABLE_TEST and, for the
managed roles, DEPLOYMENT_ENVIRONMENT, which each job's env block already sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gration job

The dynamics migration tests' engine fixture now checks that the target is the
disposable database before upgrading it, as the cliff impacts fixture does, so
without one the tests are skipped instead of migrating whatever is configured.
The integration workflow test now also requires that those migration tests
stay out of that job, and says why in its docstring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store one cliff result per simulation run in the v2 database, with grants

1 participant